COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle using Deep Reinforcement Learning

Path Following and Collision Avoidance, be it for unmanned surface vessels or other autonomous vehicles, are two fundamental guidance problems in robotics. For many decades, they have been subject to academic study, leading to a vast number of proposed approaches. However, they have mostly been treated as separate problems, and have typically relied on non-linear first-principles models with parameters that can only be determined experimentally. The rise of Deep Reinforcement Learning (DRL) in recent years suggests an alternative approach: end-to-end learning of the optimal guidance policy from scratch by means of a trial-and-error based approach. In this article, we explore the potential of Proximal Policy Optimization (PPO), a DRL algorithm with demonstrated state-of-the-art performance on Continuous Control tasks, when applied to the dual-objective problem of controlling an underactuated Autonomous Surface Vehicle in a COLREGs compliant manner such that it follows an a priori known desired path while avoiding collisions with other vessels along the way. Based on high-fidelity elevation and AIS tracking data from the Trondheim Fjord, an inlet of the Norwegian sea, we evaluate the trained agent's performance in challenging, dynamic real-world scenarios where the ultimate success of the agent rests upon its ability to navigate non-uniform marine terrain while handling challenging, but realistic vessel encounters

COLREG-Compliance for Autonomous Surface Vehicles using Deep Reinforcement Learning

Bruken av og forskning innen autonome systemer har økt kraftig i senere år, inkludert i marin sektor. Ettersom transportsektoren samtidig gjennomgår en omfattende elektrifisering, lover autonom skipsfart ikke bare reduserte kostnader gjennom nedbemanning og mer effektiv drift, men også reduserte utslipp. Helautonomi kan derfor sies å være et fremtidig mål, selv om det i dag kreves konstant monitorering av delvis autonome skip. Et av de største hindrene for å nå dette målet er utviklingen av et robust og pålitelig kontrollsystem som er i stand til å takle alle mulige situasjoner og vær. Videre er det essensielt at alle skip følger internasjonale regler for kollisjonsunngåelse på havet (engelsk forkortelse: COLREGs), slik at samarbeidet med kapteiner og andre mennesker er trygt. Siden COLREGs ble skrevet for mennesker, er de ofte formulert på tvetydig vis, og dermed ikke lett overførbare til eller verifiserbare i en digital kontekst. Grunnet disse utfordringene er det teknisk krevende å nå målet kun ved bruk av klassiske og modell-baserte metoder. Kunstig intelligens kan approksimere beslutningsmodeller, og virker derfor lovende. Forsterkende læring (engelsk: reinforcement learning) har vist et spesielt stort potensiale i et bredt spekter av applikasjoner, inkludert de som krever kontinuerlig tilstands- og handlingsrom. Siden forsterkende læring i tillegg er en selvlærende og modellfri metode er det en spesielt god kandidat for autonome skip. I denne masteroppgaven undersøkes potensialet for å flette COLREGs inn i en kontroller basert på dyp forsterkende læring (engelsk forkortelse: DRL). For å oppnå dette sammenliknes en kvalitativ og en risiko-basert metode. Begge metodene fører til gode resultater i testscenarioer, og følger COLREG-regler relevante i et miljø med én aktiv agent (regler 14-16). Dette betyr at, i tillegg til å oppnå svært god stifølging og kollisjonsunngåelse i møte med statiske objekter, var agentene i stand til å forholde seg til de implementerte COLREG-reglene. I begge tilfeller var det tydelig at en modulær funksjon for belønning fungerer godt i applikasjoner hvor agenten skal oppnå ulike konkurrerende mål. Den vellykkede inkluderingen av viktige COLREG-regler i et DRL-basert system for stifølging og kollisjonsunngåelse vitner om at DRL er gunstig for autonom navigasjon på havet

Risk-based implementation of COLREGs for autonomous surface vehicles using deep reinforcement learning

Autonomous systems are becoming ubiquitous and gaining momentum within the marine sector. Since the electrification of transport is happening simultaneously, autonomous marine vessels can reduce environmental impact, lower costs, and increase efficiency. Although close monitoring is still required to ensure safety, the ultimate goal is full autonomy. One major milestone is to develop a control system that is versatile enough to handle any weather and encounter that is also robust and reliable. Additionally, the control system must adhere to the International Regulations for Preventing Collisions at Sea (COLREGs) for successful interaction with human sailors. Since the COLREGs were written for the human mind to interpret, they are written in ambiguous prose and therefore not machine-readable or verifiable. Due to these challenges and the wide variety of situations to be tackled, classical model-based approaches prove complicated to implement and computationally heavy. Within machine learning (ML), deep reinforcement learning (DRL) has shown great potential for a wide range of applications. The model-free and self-learning properties of DRL make it a promising candidate for autonomous vessels. In this work, a subset of the COLREGs is incorporated into a DRL-based path following and obstacle avoidance system using collision risk theory. The resulting autonomous agent dynamically interpolates between path following and COLREG-compliant collision avoidance in the training scenario, isolated encounter situations, and AIS-based simulations of real-world scenarios. (C) 2022 The Author(s). Published by Elsevier Ltd

Autonomic dysfunction after mild acute ischemic stroke and six months after: a prospective observational cohort study

Abstract Introduction Autonomic dysfunction is prevalent in ischemic stroke patients and associated with a worse clinical outcome. We aimed to evaluate autonomic dysfunction over time and the tolerability of the head-up tilt table test in an acute stroke setting to optimize patient care. Patients and method In a prospective observational cohort study, patients were consecutively recruited from an acute stroke unit. The patients underwent heart rate and blood pressure analysis during the Valsalva maneuver, deep breathing, active standing, and head-up tilt table test if active standing was tolerated. In addition, heart rate variability and catecholamines were measured. All tests were performed within seven days after index ischemic stroke and repeated at six months follow-up. Results The cohort was comprised of 91 acute stroke patients, mean (SD) age 66 (11) years, median (IQR) initial National Institute of Health Stroke Scale 2 (1–4) and modified Ranking Scale 2 (1–3). The head-up tilt table test revealed 7 patients (10%) with orthostatic hypotension. The examination was terminated before it was completed in 15%, but none developed neurological symptoms. In the acute state the prevalence of autonomic dysfunction varied between 10–100% depending on the test. No changes were found in presence and severity of autonomic dysfunction over time. Conclusion In this cohort study of patients with mild stroke, autonomic dysfunction was highly prevalent and persisted six months after index stroke. Head-up tilt table test may be used in patients who tolerate active standing. Autonomic dysfunction should be recognized and handled in the early phase after stroke